Method, device, and system for providing an interface based on an interaction with a terminal

ABSTRACT

The present application discloses methods, devices, and a system for an interaction between a terminal and a user. A method includes obtaining, by one or more processors, a first input, the first input being input via a first interaction mode, providing, by the one or more processors, an interface, the interface being displayed based at least in part on the first input, obtaining, by the one or more processors, a second input, the second input being input via a second interaction mode, the second interaction mode being different from the first interaction mode, and performing, by the one or more processors, a function corresponding to the interface, the function being based at least in part on one or more of the first input, the first interaction mode, the second input, and the second interaction mode.

CROSS REFERENCE TO OTHER APPLICATIONS

This application claims priority to People's Republic of China PatentApplication No. 201811280800.8 entitled A DATA PROCESSING METHOD,APPARATUS, AND MACHINE-READABLE MEDIUM filed Oct. 30, 2018 which isincorporated herein by reference for all purposes.

FIELD OF THE INVENTION

The present application relates to a field of computer technology. Inparticular, the present application relates to a data processing method,a data processing means, an apparatus, and a machine-readable medium.

BACKGROUND OF THE INVENTION

As computer technology develops, technology directed to the interactionbetween humans and machines has also developed rapidly. Currently, modesfor interaction between humans and machines (e.g., voice interactionmodes, gesture interaction modes, and touch interaction modes) arerelatively mature. The aforementioned modes for interaction betweenhumans and machines are applied to implement routine functions, such asfunctions for adjusting volume of a terminal, brightness of a display ofa terminal, saturation of a display of a terminal, temperature, fanspeed, and other parameters.

According to some related art, the aforementioned adjustment functionsare implemented via a voice interaction mode. As an example, in the caseof a volume adjustment function, voice commands that can be input toinvoke control of the volume include: “increase the volume a little,”“decrease the volume a little,” or “adjust the volume to volume valueX.” In response to the voice command input by the user, the volume iscorrespondingly adjusted. For example, the volume value is increased ordecreased by the adjustment increment from the current base volume or isadjusted to a defined volume (e.g., value X such as corresponding to apreset volume or a value defined in the voice command). The related artgenerally requires that the user accurately determine and/or definevolume adjustment-related information (e.g., how much is the adjustmentincrement corresponding to one voice command, determine the currentvolume value corresponding to the current volume, and determine therequired target volume value). If the above information cannot beaccurately determined, a user is required to input several commands inorder to obtain a desired volume control or desired volume level.Moreover, even if the user is able to accurately determine theaforementioned volume adjustment-related information, the user isrequired to input multiple voice commands to obtain the required targetvolume value in the context of controlling or determining the currentvolume value and/or volume adjustment increment. Accordingly, therelated art methods and system for controlling such functions using avoice command are relatively inefficient and operationally difficultfrom the perspective of user control and/or user experience.

According to some related art, the aforementioned adjustment functionsare controlled via a gesture interaction mode. As an example, thegesture interaction mode corresponds to an operational mode via whichinputs are made based at least in part on gestures. For example, apredefined set of gestures can be mapped to corresponding functions, andbased at least in part on detection of a defined input the user may beprovided with N (e.g., N being a natural number) gestures. Differentgestures can correspond to different adjustment functions. Accordingly,the user can implement different adjustment functions via inputtingdifferent gestures. For example, a single-finger clockwise circlinggesture corresponds to a volume increase function. As another example, asingle-finger counterclockwise circling gesture corresponds to a volumedecrease function. As another example, a two-finger clockwise circlinggesture corresponds to a fan speed increase function. As anotherexample, a two-finger counterclockwise circling gesture corresponds to afan speed decrease function. As another example, a three-fingerclockwise circling gesture corresponds to a temperature increasefunction. As another example, a three-finger counterclockwise circlinggesture corresponds to the temperature decrease function. The value of Ngenerally matches the number of functions (e.g., a one-to-one mapping ofgestures to functions exists). In implementations according to therelated art, N is generally greater than 6, which increases thedifficulty of remembering the gestures.

According to some related art, the aforementioned functions arecontrolled via a touch interaction mode. As an example, the touchinteraction mode corresponds to an operational mode via which inputs aremade based at least in part on touch inputs such as inputs to atouchscreen. As an example, the user navigates to (e.g., invokes) afirst settings interface by first tapping a settings icon provided onthe desktop and then searches within the settings interface for asettings option corresponding to a desired adjustment function. The usercan input an input for selecting the desired adjustment function (e.g.,the user taps an icon or link corresponding to the desired adjustmentfunction to enter (e.g., navigate to) a second settings interface). Inresponse to the second settings interface being provided on theterminal, the user inputs to the second settings interface a set oftouch and swipe inputs to invoke the adjustment function adjustment. Theuse of a touch interaction mode according to the related art inconnection with invoking a desired adjustment function requires a userto cause the terminal to switch the interface from the desktop to afirst settings interface and then subsequently to a second settingsinterface. Further such a process using the touch interaction modeaccording to the related art requires multiple tap operations by theuser. Accordingly, the related art methods and system for controllingsuch functions using a touch input are relatively inefficient andoperationally difficult from the perspective of user control and/or userexperience.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a diagram of a system for invoking a function according tovarious embodiments of the present application.

FIG. 2 is a flowchart of a data processing method according to variousembodiments of the present application.

FIG. 3 is a flowchart of a data processing method according to variousembodiments of the present application.

FIG. 4 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

FIG. 5 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

FIG. 6 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

FIG. 7 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

FIG. 8 is a flowchart of a data processing method according to variousembodiments of the present application.

FIG. 9 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

FIG. 10 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

FIG. 11 is a functional diagram of a computer system for data processingaccording to various embodiments of the present application.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

To make the above-described objectives, features, and advantages of thepresent application plainer and easier to understand, the presentapplication is explained in further detail below in light of thedrawings and specific embodiments.

The technical schemes in the embodiments of the present application willbe clearly and completely described in light of the drawings in specificembodiments of the present application. Obviously, the describedembodiments are merely some of the embodiments of the presentapplication and not all the embodiments. All other embodiments obtainedby persons with ordinary skill in the art based on the embodiments ofthe present application fall within the scope of protection of thepresent application.

The concepts of the present application may easily undergo variousmodifications and substitutions, and its specific embodiments havealready been shown in the drawings and in detailed descriptions herein.However, please note that the purpose of the content described above isnot to limit the concepts of the present application to the disclosedspecific forms. Rather, the intention of the description and theattached claims of the present application is to cover all modified,equivalent, and substituted forms.

“An embodiment,” “embodiments,” “a specific embodiment” and other suchwording in this description indicate that the described embodiment mayinclude specific features, structures, or characteristics. However, eachembodiment may include or may not necessarily include these particularfeatures, structures, or characteristics. In addition, such phrases donot necessarily refer to the same embodiment. In addition, wherespecific features, structures, or characteristics are described inconnection to an embodiment, it may be assumed that, whether or not theyare described explicitly, such features, structures, or characteristicsare related to other embodiments within the scope of knowledge ofpersons skilled in the art. In addition, please understand that theitems in a list taking the form of “at least one of A, B and C” mayinclude the following possible items: (A), (B), (C), (A and B), (A andC), (B and C), or (A, B and C). Likewise, items listed in the form of“at least one of A, B or C” may imply: (A), (B), (C), (A and B), (A andC), (B and C), or (A, B and C).

In some cases, the disclosed embodiment may be implemented as hardware,firmware, software, or any combination thereof. A disclosed embodimentmay also be implemented as instructions that are carried or stored inone or more temporary or non-temporary machine-readable (e.g.,computer-readable) storage media. These instructions can be executed byone or more processors. Machine-readable media may be implemented asstorage devices, mechanisms, or other physical structures (e.g.,volatile or non-volatile memory, disks, or other media or other physicalstructure devices) for storing or transmitting information inmachine-readable form.

In the drawings, some structures or method features may be displayed inspecific arrangements and/or sequences. However, preferably, suchspecific arrangements and/or sequences are not necessary. Rather, insome embodiments, such features may be laid out in ways and/or sequencesthat differ from what is shown in the drawings. In addition, the contentof structural or method features included in a specific drawing does notimply that such features are necessary in all embodiments. Moreover, insome embodiments, these features may not be included, or they may becombined with other features.

As used herein, a “terminal” generally refers to a device comprising oneor more processors. A terminal can be a device used (e.g., by a user)within a network system and used to communicate with one or moreservers. According to various embodiments of the present disclosure, aterminal includes components that support communication functionality.For example, a terminal can be a smart phone, a server, a machine ofshared power banks, information centers (such as one or more servicesproviding information such as traffic or weather, etc.), a tabletdevice, a mobile phone, a video phone, an e-book reader, a desktopcomputer, a laptop computer, a netbook computer, a personal computer, aPersonal Digital Assistant (PDA), a Portable Multimedia Player (PMP), aMoving Picture Experts Group Audio Layer III (mp3) player, a MovingPicture Experts Group Audio Layer IV (mp4) player, a mobile medicaldevice, a camera, a wearable device (e.g., a Head-Mounted Device (HMD),electronic clothes, electronic braces, an electronic necklace, anelectronic accessory, an electronic tattoo, or a smart watch), a kiosksuch as a vending machine, a smart home appliance, vehicle-mountedmobile stations, a car, or the like. A terminal can run variousoperating systems.

As used herein, a gesture interaction mode corresponds to an operationalmode via which inputs are made based at least in part on gestures (e.g.,gestures made by a user and detected by a terminal, such as handgestures).

As used herein, a voice interaction mode corresponds to an operationalmode via which inputs are made based at least in part on a voice input(e.g., a voice command or other sound input by a user and detected by aterminal such as by a microphone associated with the terminal).

As used herein, a touch interaction mode corresponds to an operationalmode via which inputs are made based at least in part on touch inputs(e.g., inputs to a touchscreen, etc.). Other inputs can be input via atouch input. In some embodiments, a touch interaction mode includes amode according to which an input is made to a graphical user interfacesuch as an input made by a peripheral device connected to a terminal(e.g., a keyboard, a mouse, etc.).

Various embodiments provide a method, system, and/or device forprocessing data. According to various embodiments, one or more terminalsare caused to perform a function based at least in part on a pluralityof inputs that are input using a plurality of interaction modes. In someembodiments, the plurality of interaction modes include a touchinteraction mode, a voice interaction mode, a gesture interaction mode,a keyboard and/or mouse interaction mode, an interface interaction mode(e.g., an interaction mode via which inputs are made to a graphical userinterface, etc.), etc. Various other interaction modes can beimplemented.

According to various embodiments, an interface is provided at a targetposition of a screen (e.g., of a terminal or a display connected to aterminal) in response to a first input from a user. The target positioncan correspond to the hand position of the user (e.g., a position of ahand relative to one or more sensors or a field of view of the one ormore sensors). The first input corresponds to an input via a firstinteraction mode. The interface can be associated with a secondinteraction mode. For example, the second interaction mode is a mode bywhich an input is input to the interface such as via a touch input orvia a mouse or other peripheral device. In some embodiments, a functioncorresponding to the interface is executed in response to a second inputbeing obtained (e.g., from the user). The second input can correspond toan input via the second interaction mode.

Various embodiments implement a first interaction mode and a secondinteraction mode to perform a function or to cause a function to beperformed (e.g., a terminal to which the inputs are input using thevarious interaction modes communicates with another device to perform afunction or invoke a function to be performed). In some embodiments, thefirst interaction mode is different from the second interaction mode.For example, inputs to the first interaction mode are made in a mannerthat is different from a manner by which inputs to the secondinteraction mode are made. The first input corresponds to an input viathe first interaction mode. The first input via the first interactionmode is used in connection with triggering display of an interface(e.g., a graphical user interface). The second input corresponds to aninput via a second interaction mode. The first input via the firstinteraction mode is used in connection with triggering execution of(e.g., invoking) a function corresponding to the interface. Embodimentscombine the advantages of a first interaction mode and a secondinteraction mode and can avoid the defects of the first interaction modeor the second interaction mode. Embodiments implement a function (e.g.,cause a function to be performed) via a first input and a second inputand thus may reduce operational difficulty for the user while improvingoperational efficiency for the user.

According to various embodiments, function information is provided viathe interface and the function information can help the user to generatea precise second input. The function information can include a currentstatus of the function or results of execution of the function. In someembodiments, the function information is provided in response to thefirst input being input via the first interaction mode. The functioninformation can indicate functions that can be performed, results of thefirst input being obtained, etc. The function information can inform theuser to facilitate the user providing a precise second input. Therefore,the providing the function information in response to the first inputcan reduce the operational difficulty for the user by, for example,decreasing the number of human-machine interactions (e.g., human-machinevoice interactions). In some embodiments, the function information isprovided via a graphical user interface provided on a display associatedwith a terminal (e.g., a terminal to which the first input is input,etc.).

Because the interface is applicable to various functions and varioustypes of functions, different types of functions can share gestures.According to various embodiments, one or more functions are mapped to asingle gesture (e.g., various embodiments can implement a many-to-onemapping of functions to gestures). As a result, fewer gesture patternsare implemented (e.g., required) and the difficulty of learning andmemorizing gestures is reduced for users. For example differentfunctions such as functions relating to volume, brightness, saturation,contrast, temperature, and fan speed can share gestures and thus thenumber of gesture patterns can be reduced. In some embodiments, themapping of one or more functions to gestures is stored (e.g., locally ata terminal, remotely such as at a location accessible to the terminal,etc.).

Because various embodiments can quickly and directly obtain an interfacein response to obtaining a first input (e.g., from a user of aterminal), the number of operating paths used in connection withimplementing a function via a second interaction mode (e.g., a touchinteraction mode) can be decreased and thus the operational difficultyfor the user is reduced.

In some embodiments, a target position of an interface on a screen(e.g., of the terminal) corresponds to a position of a particular objector part of the user. For example, the interface target position of thescreen corresponds to a position of the user's hand. The target positioncorresponding to a position of the particular object can refer to thetarget position of the interface (e.g., a position at which theinterface is provided) in relation to the screen matches a position ofthe particular object (e.g., the user's hand) in relation to theterminal and/or to one or more sensors of the terminal. For example, theposition of the user's hand can correspond to a position of a handrelative to one or more sensors or a field of view of the one or moresensors. As an example, if the user positions the user's hand at thebottom left portion of the screen, the interface is displayed at thebottom left of the screen. According to various embodiments, theinterface is provided (e.g., on the screen of the terminal) based atleast in part on the position of the particular object. In response toobtaining the first input based on a first interaction mode, theterminal determines a position of the particular object (e.g., theuser's hand) and provides the interface based at least in part on thefirst input and/or the position of the particular object. Thepositioning of the interface based at least in part on the position ofthe particular object (e.g., the user's hand) increases relatednessbetween the user (e.g., the hand) and the interface. The positioning ofthe interface based at least in part on the position of the particularobject facilitates an experience for the user according to which theinterface is positioned wherever the hand is positioned (e.g., theposition of the interface matches the position of the user's hand).Accordingly, hand operations become more convenient for a user, and theuser experience is improved.

Various interaction modes can be implemented for the first interactionmode and the second interaction mode (e.g., according to actualapplication needs). In some embodiments, an administrator or user canconfigure the particular interaction mode corresponding to the firstinteraction mode and/or the second interaction mode. The interactionmode corresponding to the first interaction mode and/or the secondinteraction mode can be configured based on terminal settings, userpreferences, historical information pertaining to use of the terminal orinteraction with an interface, etc.

In some embodiments, the first interaction mode is a voice interactionmode, and the second interaction mode is a gesture interaction mode or atouch interaction mode. According to an example in which the firstinteraction mode is a voice interaction mode, and the second interactionmode is a gesture interaction mode or a touch interaction mode, thelengthy interaction process entailed in an interaction mode limited onlyto voice is improved. For example, the terminal invokes (e.g., displays)an interface in response to a voice input, and the user can input thesecond input quickly and conveniently via the gesture interaction modeor the touch interaction mode to implement a corresponding function(e.g., a function corresponding to the interface). The interface isprovided quickly and conveniently in response to the voice input, andthe user is able to input the second input quickly and convenientlythrough the gesture interaction mode or the touch interaction mode toimplement a function corresponding to the interface. If the secondinteraction mode corresponds to a gesture interaction mode (e.g., suchthat the second input is input via a gesture interactive operation),different types of functions can share gestures. For example, a singlegesture can be mapped to different types of functions. A function to beinvoked can be determined based at least in part on the first input andthe second input (e.g., the obtained gesture corresponding to the secondinput in combination with the particular interface being provided and/orthe first input). Because a plurality of functions can be mapped to asingle gesture, fewer gestures (e.g., gesture patterns) are used, andthus the user encounters less difficulty in learning and memorizinggestures.

In some embodiments, the first interaction mode is a touch interactionmode, and the second interaction mode is to be a gesture interactionmode or a voice interaction mode. According to an example in which thefirst interaction mode is a touch interaction mode, and the secondinteraction mode is a gesture interaction mode or a voice interactionmode, the first input can be a preset touch input. The preset touchinput can be an operation corresponding to a preset number of screentaps, an operation corresponding to a screen tap with a preset pressingforce, or a swipe operation corresponding to a preset path produced onthe screen. Various preset touch inputs can be mapped to one or morecorresponding functions. A mapping of the various preset touch inputs toone or more corresponding functions can be stored locally at theterminal or at a remote storage that is accessible to the terminal. Theone or more corresponding functions mapped to the preset touch input caninclude invoking an interface, etc. In response to receiving the firstinput, the terminal can obtain (e.g., invoke) an interface. Theinterface can be invoked based at least in part on the first input. Insome embodiments, function information is provided (e.g., displayed) viathe interface. As an example, the interface can provide the currentstatus of the function, or results of the user's execution of thefunction, etc. The function information can be used in connection withobtaining the second input. For example, the user can use the functioninformation to generate a precise second input. The terminal obtains thesecond input and performs a function based at least in part on thesecond input. The terminal can determine the function to perform inresponse to obtaining the second input, and the function to be performedcan be determined based at least in part on one or more of the firstinput, the interface, and/or the second input. Therefore, the functioninformation can reduce the operational difficulty for the user by, forexample, decreasing the number of human-machine voice interactions. Asan example according to which the first input is associated with avolume control function, the interface can provide (e.g., display) acurrent value and maximum value of the volume (e.g., a playback volumeof the terminal, etc.) to help the user with determining the requiredtarget volume value X and inputting the second input for triggering anadjustment to the target volume X (e.g., by a voice command, etc.).

In some embodiments, the first interaction is a gesture interactionmode, and the second interaction mode is a touch interaction mode or avoice interaction mode. According to an example in which the firstinteraction mode is a gesture interaction mode, and the secondinteraction mode is a touch interaction mode or a voice interactionmode, the first input can be a preset gesture input. Various presetgesture inputs can be mapped to one or more corresponding functions. Amapping of the various preset gesture inputs to one or morecorresponding functions can be stored locally at the terminal or at aremote storage that is accessible to the terminal. The one or morecorresponding functions mapped to the preset gesture input can includeinvoking an interface, etc. The gesture input can correspond to anoperation in which a particular object (e.g., the user's hand) moveshorizontally or vertically in circles or in any other direction.Movement of the particular object can be determined in relation to theterminal (e.g., a movement of the particular object relative to theterminal or one or sensors of the terminal). In response to receivingthe first input, the terminal can obtain (e.g., invoke) the interface.The interface can be invoked based at least in part on the first input.In some embodiments, function information is provided (e.g., displayed)via the interface. As an example, the interface can provide the currentstatus of the function, or results of the user's execution of thefunction, etc. The function information can be used in connection withobtaining the second input. For example, the user can use the functioninformation to generate a precise second input. The terminal obtains thesecond input and performs a function based at least in part on thesecond input. The terminal can determine the function to be performed inresponse to obtaining the second input, and the function to be performedcan be determined based at least in part on one or more of the firstinput, the interface, and/or the second input.

In some embodiments, application contexts include remote operationcontexts, close range contexts (e.g., for instances when preciseoperation is inconvenient), touchscreen contexts, etc. Various otherapplication contexts are possible. Remote operation contexts caninclude: smart television contexts and large-screen contexts. Closerange contexts can include: driving contexts.

As an example, in the case of a driving context, the user generally isrequired to focus the user's attention on driving (e.g., the userfocuses attention on the road conditions surrounding the vehicle inorder to drive safely). Therefore, the user's attention is focused ondriving and the user cannot conveniently perform precise operations.Various embodiments support the user in implementing a desired functionbased at least in part on obtaining a first input via a firstinteraction mode and one or more second inputs via a second interactionmode. The first interaction mode can be a voice interaction mode, andthe second interaction mode can be a gesture interaction mode. Becauseboth the voice interaction mode and the gesture interaction mode areinteraction modes that do not require the user to provide physicalcontact with the terminal, the use of the voice interaction mode and thegesture interaction mode in connection with invoking a function allowsthe user to focus on driving without having to direct the user'sattention elsewhere. Accordingly, such use of the use of the voiceinteraction mode and the gesture interaction mode in connection withinvoking a function can improve driving safety. In some embodiments, oneof the first interaction mode and the second interaction mode is aninteraction mode that does not require the user to provide physicalcontact with the terminal (e.g., a voice interaction mode or a gestureinteraction mode), and the other of the first interaction mode and thegesture mode is a touch interaction mode.

As an example, in the case of a terminal having a touchscreen, then in atouchscreen context, related art generally requires a user to navigatethrough multiple levels of an interface before entering the functioninterface (e.g., a desired interface via which a user can control and/orinvoke a corresponding function). According to various embodiments, theinterface is invoked in response to a first input via a firstinteraction mode (e.g., the voice interaction mode) and the desiredfunction is invoked in response to a second input via the secondinteraction mode (e.g., the gesture interaction mode). In response tothe second input, the terminal performs or causes another terminal toperform (e.g., via sending an instruction to the other terminal) thecorresponding function. Therefore, various embodiments decrease theoperating paths involved in implementing a function in comparison to thetouch interaction mode of the related art, and thus various embodimentsreduce the operational difficulty for the user.

FIG. 1 is a diagram of a system for invoking a function according tovarious embodiments of the present application.

Referring to FIG. 1, system 100 for invoking a function or controlling aterminal is provided. System 100 can implement at least part of process200 of FIG. 2, process 300 of FIG. 3, interaction 400 of FIG. 4,interaction 500 of FIG. 5, interaction 600 of FIG. 6, interaction 700 ofFIG. 7, process 800 of FIG. 8, interaction 900 of FIG. 9, and/orinteraction 1000 of FIG. 10. System 100 can be implemented at least inpart by computer system 1100 of FIG. 11.

System 100 comprises terminal 110. System 100 can further include server120 with which terminal 110 communicates. In some embodiments, system100 includes network 130 over which terminal 110 and server 120communicate.

Network 130 can be a wired network such as a local area network. In someembodiments, network 130 is a wireless network such as a cellularnetwork, a WiFi network, a satellite communication network, a wirelesslocal area network, etc.

According to various embodiments, terminal 110 obtains a first input viaa first interaction mode. As an example, a user inputs the first input(e.g., a voice input, a gesture, and/or a touch input) to terminal 110.In response to receiving the first input, terminal 110 obtains aninterface based at least in part on the first input. As an example, theinterface is obtained from local storage of terminal 110 based at leastin part on the first input. Terminal 110 can store a mapping of inputsto interfaces, and terminal 110 can determine the interface based atleast in part on the first input and the mapping of inputs tointerfaces. As another example, the interface is generated based atleast in part on the first input. Terminal 110 can generate theinterface based at least in part on the first input and one or moretemplates and/or protocols for generating the interface. The one or moretemplates and/or protocols can be stored locally at terminal 110 and/orstored at server 120.

In some embodiments, terminal 110 determines the interface and/orobtains the interface 110 based at least in part on information thatterminal 110 receives from server 120. For example, in response toreceiving the first input, terminal 110 provides to server 120information pertaining to the first input (e.g., an indication that thefirst input is received, a corresponding type of input, a commandassociated with the first input, etc.). Server 120 can provide toterminal 110 the interface or information pertaining to the interface(e.g., information from which the interface is determined, and/orinformation from which the interface is generated).

In response to obtaining the interface (e.g., obtaining the interfacefrom storage, determining the interface, generating the interface,etc.), terminal 110 provides the interface. For example, terminal 110displays the interface on a display (e.g., a touchscreen). Terminal 110can obtain a second input via a second interaction mode. For example,the user inputs the second input to terminal 110 via the secondinteraction mode. According to various embodiments, the secondinteraction mode is different from the first interaction mode. Thesecond input can be input with respect to the interface. For example,the second input can correspond to an interaction with the interface(e.g., by the user).

In response to obtaining the second input, terminal 110 performs acorresponding function. In some embodiments, terminal 110 stores amapping of inputs to functions (or has remote access to such a mapping)and terminal 110 determines the function corresponding to the secondinput based at least in part on the mapping of inputs to functions. Insome embodiments, the function performed in response to the second inputis based at least in part on the second input and one or more of theinterface, the first input, the first interaction mode, and the secondinteraction mode. In some embodiments, a corresponding function is to beperformed at server 120. For example, in response to receiving thesecond input, terminal 110 provides server 120 with an instructioncorresponding to the second input to cause server 120 to perform acorresponding function (e.g., store information, provide alocation-based service to terminal 110, provide information to terminal110, provide information to another terminal connected to network 130,and/or cause another terminal connected to network 130 to perform afunction).

In some embodiments, terminal 110 and/or server 120 runs a client. Theclient can provide an application context such as the applicationcontext in which the first input and/or the second input is obtained.For example, the client may be an application (app), such as anavigation app, an e-commerce app, an instant messaging app, an inputmethod app, a weather app, a media playback app, a social media app, oran app that comes with an operating system, running on a terminal, etc.Terminal 110 and/or server 120 can run various apps which can providethe application context according to which the first input and/or thesecond input is obtained.

In some embodiments, terminal 110 and/or server 120 includes a built-inor externally-connected screen. The screen is configured to displayinformation such as the interface that is invoked in response to thefirst input via the first interaction mode.

Terminal 110 can correspond to, or comprise a vehicle-mounted device.Examples of vehicle-mounted devices include: head-up displays (HUDs). AHUD is generally installed in front of the driver. The HUD can providethe driver of the vehicle with various trip information such as vehiclespeed, fuel consumption, navigation, vehicle information such as astatus of one or more characteristics of the vehicle (e.g., a status ofone or more doors, an air pressure warning for a tire, an alerttriggered by information obtained by one or more sensors connected tothe vehicle, etc.), and paired mobile device information such as evenincoming phone calls and message notifications. The HUD can provide suchinformation during driving. In other words, the HUD can integrate manyfunctions into one unit and thus help the driver to concentrate on roadand traffic conditions.

According to various embodiments, the functions performed by theterminal (e.g., in response to the second input being obtained) caninclude, for example, system functions and/or application functions.System functions can include parameter setting functions such assettings relating to volume (e.g., volume of media playback, volume ofalerts, etc.), brightness (e.g., brightness of the display on thevehicle, brightness on the mobile device paired to a vehicle, etc.),saturation, indicator parameters, contrast, temperature, and fan speed.Various other parameter setting functions can be implemented.Application functions can include functions internal to an applicationprogram. An application program can determine the functions according toactual application need. An example of a function determined accordingto an application need includes a display parameter setting function fora browser application. An application function can be a function forobtaining information pertaining to the application (e.g., to display amenu, to play back a desired media file, to obtain information from aserver such as location-based service information, etc.).

According to various embodiments, the functions in performed by theterminal (e.g., in response to the second input being obtained) aresystem-wide functions, such as parameter adjusting functions relating toPC volume, brightness, and backlighting or to vehicle-mounted devicevolume, fan speed, or temperature. In other words, various embodimentscan adjust or control volume, fan speed, temperature, or another systemfunction for a vehicle-mounted device. In some embodiments, the terminaladjusts or controls volume, brightness, saturation, or another systemfunction for a smart television set. In some embodiments, the terminaladjusts or controls volume, brightness, or another system function for alarge screen such as in a PC or tablet. In some embodiments, theterminal adjusts or controls volume, brightness, or another systemfunction for a mobile device.

Parameters that are controlled or adjusted based at least in part on asystem-wide function (e.g., a parameter adjusting function) such as inresponse to an input by a user can include operating parameters ofcomponents connected to (or implemented by) the terminal. The connectionof the components to the terminal can include: non-contact connections(such as network connections) and/or contact connections. Volume in thiscontext can refer, for example, to the operating parameter of thesound-producing component of a speaker (e.g., a speaker of a vehicle, aspeaker of a mobile device, a Bluetooth speaker connected to theterminal, etc.). Brightness, saturation, and contrast can refer, forexample, to the operating parameters of a screen display of a terminal.Temperature and fan speed can refer, for example, to the operatingparameters of an air-conditioner. As an example, in a driving context,the operating parameters of a vehicle-mounted air-conditioner can beadjusted or controlled via a vehicle-mounted device. As an example, in ahousehold context, operating parameters of a home air-conditioner can beadjusted or controlled through a smart television set or a smart phone.Various other operating parameters can be adjusted and/or controlled(e.g., based at least in part on one or more inputs).

Various embodiments include a gesture recognition apparatus to obtain auser gesture and recognize the obtained gesture. The gesture recognitionapparatus can include one or more sensors. The one or more sensors candetect a gesture and provide corresponding information to a processorthat processes the information to determine the gesture. Accordingly,the terminal can respond (e.g., perform one or more functions inresponse) to the gesture recognition result. The gesture recognitionincludes detecting a gesture, determining a type of gesture, and/ordetermining context information pertaining to a context of the terminalif the gesture is obtained.

According to various embodiments, an orientation of a gesture includesone or more of direction, angle, shape, and arc. Various gestures can bedetected or obtained by the terminal. For example, in an implementation,a diverse set of gestures can be obtained via a hand in a preset statemoving in any direction, circling a point on a plane, or rotating alongan axis. Preset states of a hand in connection with obtaining a gesturecan include a fist, an open palm, a half-open palm, a particularconfiguration of fingers extended, etc. Various gestures can be definedand/or mapped to various functions. A user or administrator can definegestures. Defining a gesture can include characterizing the orientationof the gesture. The definition of a gesture can be modified based onhistorical information such as via use of a learning or trainingalgorithm that defines the gesture based in part on past gesturesobtained from the user (e.g., from a same user, or from differentusers). A gesture can be used and/or defined according to actualapplication (e.g., a context or implementation).

According to various embodiments, the gesture recognition includes oneor more of: at least two infrared cameras, an image-capturing module,infrared light-emitting diodes, an infrared light-emitting diode driver,a calculating module, and a gesture recognition module. Theimage-capturing module is electrically connected to the two infraredcameras. The infrared light-emitting diode driver is electricallyconnected to the infrared light-emitting diodes and configured to drivethe infrared light-emitting diodes to emit light. The calculating moduleis electrically connected to the image-capturing module, the infraredlight-emitting diode driver, and the gesture recognition module. Theimage-capturing module captures gesture image data through the twoinfrared cameras and provides the gesture image data to the calculatingunit for processing. The calculating module obtains the gesture imagedata and, after subjecting the gesture image data to three-dimensionalcoordinate positioning, provides the resulting image data to the gesturerecognition module. The gesture recognition module recognizes (e.g.,determines) the gesture via pattern matching. For example, a mapping ofgesture information (e.g., processed image data corresponding togestures) to gestures is queried to determine the gesture that isobtained.

The gesture recognition apparatus can be independent of, or integratedinto, the terminal 110 (e.g., the terminal used to implement variousembodiments). Embodiments of the present application impose norestrictions as to specific gesture recognition apparatuses or as to thespecific relationship between a gesture recognition apparatus and theterminal used to implement a method according to embodiments of thepresent application.

FIG. 2 is a flowchart of a data processing method according to variousembodiments of the present application.

Referring to FIG. 2, process 200 for processing data is provided.Process 200 can implement invocation of a function or control of aterminal. Process 200 can be implemented in connection with at leastpart of process 300 of FIG. 3, interaction 400 of FIG. 4, interaction500 of FIG. 5, interaction 600 of FIG. 6, interaction 700 of FIG. 7,process 800 of FIG. 8, interaction 900 of FIG. 9, and/or interaction1000 of FIG. 10. Process 200 can be implemented at least in part bysystem 100 of FIG. 1 and/or computer system 1100 of FIG. 11.

At 210, a first input is obtained. In some embodiments, a terminalreceives the first input from a user (e.g., based on a user interactingwith the terminal). The first input can be one or more of a touch input,a voice input, a gesture, an input to a peripheral device, etc. Thefirst input is obtained via a first interaction mode. The firstinteraction mode can be configured based on terminal settings, userpreferences, historical information pertaining to use of the terminal orinteraction with an interface, etc.

According to various embodiments, the first interaction mode is a touchinteraction mode, a voice interaction mode, a gesture interaction mode,a keyboard and/or mouse interaction mode, an interface interaction mode(e.g., an interaction mode via which inputs are made to a graphical userinterface, etc.), etc. Various other interaction modes can beimplemented. The first interaction mode can be determined based at leastin part on the first input. For example, the first interaction mode canbe determined based on a type of input associated with the first input(e.g., whether the first input is a voice command, a touch input, agesture, etc.). The terminal can receive the first input and determinethat the first input corresponds to an input mapped to a particularinteraction mode. The terminal can determine that the first interactionmode corresponds to the particular interaction mode associated with thefirst input. In some embodiments, the terminal operates according to thefirst interaction mode before the first input is received.

In some embodiments, the first input is used to trigger display of aninterface. For example, the first input can be mapped to a function thatinvokes an interface, and in response to the terminal obtaining thefirst input, the terminal invokes and displays the interface based onthe first input in connection with a query of the mapping of inputs tofunctions.

Various interaction modes can be implemented as the first interactionmode. The interaction mode to be used as the first interaction mode canbe set by a user (e.g., selected by a user, or set according to userpreferences, etc.), an administrator or a developer associated with theterminal, or the application corresponding to the application context inwhich the first input is obtained.

According to various embodiments, the first interaction mode is a voiceinteraction mode, and the first input is a voice command. The voicecommand can characterize (e.g., describe) the function to be performed(e.g., the invocation of the interface, etc.). The voice command caninclude information describing the function to be performed. The voicecommand can include an identifier (e.g., ID) of the function to betriggered (e.g., invoked). For example, the voice command includes thefunction name of the function to be performed. The voice command caninclude information indicating a type of function to be performed and/ora parameter associated with the function to be performed.

In some embodiments, the first interaction mode is a gesture interactionmode. The first input can be a hand operation (e.g., that is input bythe user in relation to one or more sensors). The hand operation caninclude movement in any direction such as horizontal, vertical, orcircling.

In some embodiments, the first interaction mode is a touch interactionmode. The first input can be an operation including a preset number ofscreen taps, an operation including a screen tap with a preset pressingforce, a swipe operation including a preset path produced on the screen,etc. The swipe operation can include a swipe having one or morecharacteristics including direction, length, location at which swipebegins, location at which swipe ends, etc. The preset pressing force maybe obtained through 3D touch technology.

At 220, an interface is provided. In some embodiments, the terminalprovides the interface. For example, the terminal displays the interfaceon a display (e.g., a touchscreen). According to various embodiments,the interface is provided based at least in part on the first input. Forexample, in response to receiving the first input, the terminal obtainsthe interface. The interface can be preset or preconfigured before theterminal receives the first input, or the interface can be configured orgenerated based at least in part on the first input. For example, theinterface can be configured based on a particular command associatedwith the first input, a type of input corresponding to the first input,a type of command associated with the first input, a first interactionmode via which the first input is received, etc.

According to various embodiments, the interface is provided at a targetposition in relation to a screen. The target position corresponds to aposition on the screen at which the interface is displayed. For example,the interface is provided at a bottom left corner of the screen, a topleft corner of the screen, a middle of the screen, a top right corner ofthe screen, a bottom right corner of the screen, etc.

In some embodiments, a target position corresponds to a position of aparticular object such as the position of a hand of the user (e.g., aposition of a hand relative to one or more sensors or a field of view ofthe one or more sensors). For example, the position of the user's handcan correspond to a position of a hand relative to one or more sensorsor a field of view of the one or more sensors. According to variousembodiments, the interface is provided (e.g., on the screen of theterminal) based at least in part on the position of the particularobject. In response to obtaining the first input based on a firstinteraction mode, the terminal determines a position of the particularobject (e.g., the user's hand) and provides the interface based at leastin part on the first input and/or the position of the particular object.Thus, relatedness between the hand and the interface is increased, andthe user experiences an effect in which the interface is locatedwherever the user's hand is located. As a result, hand operations becomemore convenient, and the user experience is improved.

In some embodiments, the hand position is the operation-triggeringposition. As an example, the operation-triggering position correspondsto a location (e.g., position) at which a touch input is input to atouchscreen (e.g., a position at which the hand touches the touchscreen,a position at which a stylus touches the touchscreen, etc.). As anotherexample, the operation-triggering position corresponds to a location atwhich a hand (or other input element such as a stylus, a finger, etc.)hovers in front of the screen. In some embodiments, theoperation-triggering position corresponds to a position associated witha voice input. For example, a location of a source of a voice input(e.g., a user) can be determined and deemed to be the operating positionassociated with the voice input. As an example, if the source of thevoice input is located at “center of the screen,” then the center of thescreen is determined to be the operation-triggering position. Accordingto various embodiments, the operation-triggering position is determinedbased at least in part on a real-time instruction or a setting obtainedfrom the operating system or another app. For example, theoperation-triggering position is based on screen coordinate informationoutput by the operating system or another app.

In some embodiments, the interface corresponds to a second interactionmode. For example, the interface is configured and/or provided inconnection with receiving one or more inputs via the second interactionmode. According to various embodiments, the second interaction mode isdifferent from the first interaction mode. The second interaction modecan be determined (e.g., selected) based at least in part on the firstinput and/or an application context. For example, the second interactionmode is determined based at least in part on a context of a terminal. Ifthe terminal is a mobile terminal paired with a vehicle or if theterminal is a vehicle, the context of the terminal can be the vehiclemoving, the vehicle in a parked state, etc. The context of the terminalcan be determined based on whether a particular application is running,whether a particular application is in a foreground of the terminal,etc.

In some embodiments, the terminal provides an indication of the secondinteraction mode. For example, the terminal provides an indication ofthe manner by which the user is to interact with the interface and/orthe terminal. The terminal can display an indication of the secondinteraction mode and/or display an indication of how the user is tointeract with the interface and/or the terminal (e.g., a manner by whichthe user can invoke a corresponding function). The terminal can providean audio prompt indicating the second interaction mode, etc.

In some embodiments, the interface includes function information. Thefunction information can include one or more of the following types ofinformation: a current status of the function, operation guidanceinformation (e.g., a user manual, etc.) corresponding to the function,and user execution results for the function. The function informationcan be used in connection with generating a precise second input. Forexample, the function information helps the user to generate a precisesecond input. Therefore, the function information can reduce theoperational difficulty for the user by, for example, decreasing thenumber of human-machine voice interactions.

In some embodiments, the interface includes a widget. The widget caninclude a control element that is configured to respond to a slideevent. For example, an input can be input to the touchscreen viatoggling the control element of the widget with a touch input (e.g., aslide input). The widget can include: a guide container in which thecontrol element is caused to slide. As an example, the guide containercorresponds to an element such as a slider window and the controlelement is the button inside the slider window. The position of thecontrol element in the guide container can characterize the currentstatus of the function. For example, the guide container can have a leftside corresponding to an “off” status of the corresponding function, anda right side corresponding to an “on” status of the correspondingfunction. If the control element is toggled to the left side of theguide container, the current status of the corresponding function isoff, and if the control element is toggled to the right side of theguide container, the current status of the corresponding function is on.

In some embodiments, the position of the control element of the widgetmatches (e.g., corresponds to) the hand position of the user. Theposition of the control piece can be configured to correspond to aposition at which the touch input is input to the touchscreen (e.g., aposition at which the hand touches the screen). The position of thecontrol element can correspond to the position projected onto the screenfrom the hand above the screen (e.g., a position at which the handhovers above (or in front of) the screen). The correspondence betweenthe position of the control element and the user's hand (or other inputmechanism such as a stylus) enables the user to easily generate thesecond input.

At 230, a second input is obtained. In some embodiments, a terminalreceives the second input from a user (e.g., based on a user interactingwith the terminal). The second input can be one or more of a touchinput, a voice input, a gesture, an input to a peripheral device, etc.The second input is obtained via a second interaction mode. The secondinteraction mode can be configured based on terminal settings, userpreferences, historical information pertaining to use of the terminal orinteraction with an interface, etc. The second interaction mode can bebased at least in part on one or more of the first input, the firstinput interaction mode, an application context, a context of theterminal, etc.

In some embodiments, the second interaction mode is preset such.Accordingly, in response to the first input being received via the firstinteraction mode, the second interaction mode is configured. Forexample, the second interaction mode is determined based at least inpart on the first input and/or the first interaction mode.

In some embodiments, the second interaction mode is determined inresponse to receiving the second input and in response to determiningthat the second input corresponds to a preset input according to anyinteraction mode that is different from the first interaction mode.

According to various embodiments, the second input is associated with anelement (e.g., a link, a button, etc.) provided on the interface and/orinformation provided on the interface.

According to various embodiments, the interface is displayed before orafter the second input is obtained.

According to various embodiments, if the first interaction mode is avoice interaction mode, the second interaction mode is determined to bea touch interaction mode if a touch input is input to the touchscreen(e.g., if the hand of the user touches said screen, etc.). For example,if the user issues a voice command and taps any position on the screen(e.g., with the user's hand), the position at which the screen is tappedserves as the basis to determine the target position for displaying aninterface. The target position can be the position at which the handtouches the screen.

According to various embodiments, if the first interaction mode is avoice interaction mode, the second interaction mode is determined to bea gesture interaction mode if an input is received without a touch tothe touchscreen (e.g., the input is not a touch input such as if agesture input is received and the hand of the user does not touch saidtouchscreen). For example, if the user issues a voice command and causesthe user's hand to hover over the screen, the hand-hovering positionserves as the basis for determining the target position for displayingan interface. The target position can be the projection position of thehand on the screen.

At 240, a function is performed. In some embodiments, the function isperformed in response to the second input being obtained (e.g., via thesecond interaction mode). As an example, the function corresponds to thesecond input and/or the second interaction mode. The terminal can storea mapping of inputs to functions, and the terminal can determine thefunction corresponding to the second input based at least in part on themapping of inputs to functions. In some embodiments, the functionperformed in response to the second input is based at least in part onthe second input and one or more of the interface, the first input, thefirst interaction mode, and the second interaction mode.

In some embodiments, a client (e.g., an app running on the terminal)performs at least part of process 200 of FIG. 2.

Process 200 of FIG. 2 can be performed while the device is in any state.In some embodiments, process 200 is performed when the terminal is in aparticular corresponding state (e.g., a particular application context,a particular operating context, a particular environmental context,etc.). For example, the terminal, while displaying any page (or runningany application), performs 220 in response to the obtaining of the firstinput at 210, and then the terminal performs 240 in response to thesecond input at 230.

According to various embodiments, a target function is determinedcorresponding to the first input. For example, 210 and/or 220 comprisesdetermining the target function corresponding to the first input. Inresponse to the target function being determined, the interface isconfigured and/or determined based at least in part on the targetfunction. For example, the interface corresponding to the targetfunction is displayed at the target position on the screen. According tovarious embodiments, different target functions correspond to the sameor different interfaces. An interface corresponding to the targetfunction can include: the widget such as a widget described above and atarget function identifier (ID). The target function ID can include: thename, an icon, etc. The position of the control element in the guidecontainer can be determined based at least in part on the currentparameter values corresponding to the target function.

At 240, a function corresponding to the interface is performed inresponse to the second input. For example, the second input correspondsto a touch input, and a corresponding to the interface is triggered andexecuted. As another example, if the interface corresponds to a volumecontrol interface, and the second input corresponds to a touch input,and a volume control is triggered and executed.

The second input can be a sliding of the control element in a widgetprovided on the interface. For example, if the interface widget includesa control element and a guide container, the user can input the secondinput in a manner that controls the position of the control elementwithin the guide container. An example of the second input can be aslide input with respect to the control element. In some embodiments,the direction of the guide container matches the slide direction of thecontrol piece. For example, the guide container can be an oblong, andthe slide direction can be horizontal left to right or right to left. Asanother example, the guide container is a rectangle having a longer sideconfigured to run horizontally. According to various embodiments, theslide direction is based at least in part on the guide container and canbe any direction, such as horizontal, vertical, up-down, or left-right.The second input obtained at 230 can correspond to an input with theslide direction. The implementation of a control element and a guidecontainer can reduce the difficulty for a user to learn and memorize thesecond input and can improve second input accuracy.

As an example, if the screen is parallel to the face of the user (e.g.,if the user is looking to the screen), the directions of the guidecontainer include: the left-right direction (e.g., the guide containeris an oblong shape with longer dimension extending horizontally).According to such an example, the gesture corresponding to the secondinput can thus be a swipe in a left-right direction. The second inputcan also be a voice command corresponding to an instruction to slide thecontrol element in a particular direction (or from one point to anotherpoint). For example, a gesture direction that extends from left to rightcan cause the control element to slide from left to right. As anotherexample, a gesture direction that extends from right to left can causethe control piece to slide from right to left. Of course, the gesturedirection corresponding to the second input can differ from the slidedirection. For example, a gesture direction that goes from left to rightmay cause the control piece to slide from up to down. The movement ofthe control element within (or along) the guide container can control acharacteristic of the terminal or of a service. For example, themovement of the control element can change the volume (e.g., a slide ofthe control element in a rightwards direction can progressively increasea volume of playback, a slide of the control element in a leftwardsdirection can progressively decrease a volume of playback, etc.).

The widget described above (e.g., that includes a control element) ismerely an optional embodiment of an interface. The control element canbe implemented in various forms (e.g., a button, a checkbox, a knob thatcan be turned, etc.). In some embodiments, the interface includes aplurality of function options. The plurality of function options can bearranged sequentially. As an example, the plurality of function optionscan be implemented in connection with a plurality of control elementseach of which corresponds to a different one of the plurality offunction options. The selected function option (e.g., of the pluralityof function options) serves as the target function. According to variousembodiments, the target function is selected based at least in part onthe second input. For example, the function corresponding to thefunction option selected by (or triggered by) the second input is deemedthe target function. As an example, volume is differentiated into aplurality of function options in order of high to low. The plurality offunction options can include: function option 1, function option 2, . .. function option N (where N is a natural number greater than 1). Thus,the user can perform an adjustment on one of the target function optionsvia inputting the second input.

In some embodiments, the control element or target function optionvaries with the second input. For example, the control element or thetarget function option can vary based at least in part on the secondinput (e.g., a type of second input or an interaction mode via which thesecond input is input). As an example, if the second input is touchinput, the control element moves synchronously (or contemporaneously)with a hand position of the user's hand. As another example, if thesecond input is a voice input or a gesture input, a single voice inputor the gesture input corresponding to a time period T can be determinedand a mapping relationship between the single voice input or gestureinput and the slide unit of the control element or target functionoption can be determined. The control element or target function optioncan thereby be updated based at least in part on the single voice inputor the gesture input.

In some embodiments, the unit measuring the amount of sliding of thecontrol element (also referred to as the slide unit) corresponding to asingle voice input is Δa. Accordingly, a shift of Δa in the controlelement or the target function option (e.g., an amount a volume ischanged based on a single input) can be performed based at least in parton the single voice input. The single voice input can indicate adirection of the shift of Δa to be implemented with respect to thecontrol element or the target function option. For example, in the caseof the corresponding function being volume controlled, a voice inputindicating that the volume is to be increased can cause the controlelement or the target function option to be shifted in a single slideunit of Δa (e.g., the volume can be increased by an amount correspondingto Δa).

In some embodiments, the slide unit corresponding to a gesture inputwithin the time period T is a shift of Δb. Accordingly, a shift oft×Δb/T can be performed with respect to the control piece or the targetfunction option based at least in part on the slide direction and time tof the gesture input.

In some embodiments, a unit according to which a characteristic of theterminal (e.g., a volume, a brightness, etc.) is changed can be set andthe change in the corresponding characteristic can be in increments ofthe unit based at least in part on the second input.

According to various embodiments, if the second interaction mode is agesture interaction mode, the performing the function (e.g., in responseto the second input) such as the specific function corresponding to theinterface includes performing the target function corresponding to afunction-selecting gesture in response to the function-selecting gesture(e.g., [what?] being input by the user).

The function-selecting gesture can be associated with invoking executionof the function corresponding to the interface. For example, a gesturedirection that goes from left to right causes the control element toslide from left to right and thereupon to implement an “adjust up”function. As another example, a gesture direction that goes from rightto left causes the control element to slide from right to left andthereupon to implement an “adjust down” function.

In some embodiments, the orientation of the function selecting gesture(or the control element) can be based at least in part on the positionat which the function information is provided (e.g., displayed). As anexample, the terminal stores a mapping of the display positioncorresponding to the function information and the orientation of thefunction-selecting gesture. Accordingly, the displayed functioninformation can indicate (or be indicative) to the user of theorientation of the function-selecting gesture. The relationship betweenthe orientation of the function selecting gesture (or the controlelement) and the function information or the position at which thefunction information is provided can further reduce the difficulty forthe user to learn and memorize the gesture associated with invoking adesired function while increasing the convenience of gestureinteractions.

The orientation of the function-selecting gesture can include direction.The display position corresponding to the function information can beused to indicate the direction of the function-selecting gesture. Forexample, the display position corresponding to the function informationis provided on the left side of the widget. According to such anexample, the direction of the function-selecting gesture is left (e.g.,a leftward shift or a leftward rotation). As another example, thedisplay position corresponding to the function information is providedon the right side of the widget. According to such an example, thedirection of the function-selecting gesture is right (e.g., a rightwardshift or a rightward rotation).

According to various embodiments, the display position corresponding tothe function information is mapped to the orientation of thefunction-selecting gesture. In some embodiments, determining a mappingof display position corresponding to the function information to theorientation of the function-selecting gesture includes determining thedisplay position corresponding to the orientation of thefunction-selecting gesture based at least in part on the orientation ofthe function-selecting gesture and the mapping relationship between thefunction and the function-selecting gesture. As an example, if Function1 corresponds to Function-selecting Gesture 1, then Display Position Bcorresponding to Function 1 information can be determined based onOrientation A of Function-selecting Gesture 1, wherein Orientation A andDisplay Position B match each other. For example, if Orientation A is aleftward shift direction, Display Position B may be a position on theleft side of the screen.

In some embodiments, the function information is located on the edge ofthe widget. The edge can include at least one of the following: an upperedge, a lower edge, a left edge, and a right edge. Position informationcorresponding to the edge can indicate to the user the orientation ofthe function-selecting gesture.

In an embodiment of the present application, the function informationmay include: text and/or icons. The text can correspond to a functionkeyword (e.g., the name of the function, a description of the function,etc.). The icon can be an icon that is used to refer to the meaning ofthe function. For example, the icon for the “adjust up” function icon is“→”, and the “adjust down” function icon is “←”.

In some embodiments, display of the interface is stopped in response toa context and/or in response to one or more conditions being satisfied.For example, display of the interface occurs in response to adetermination that a hand action by the user satisfies a precondition.As another example, display of the interface occurs in response to adetermination that a hand gesture corresponds to a gesture that ismapped to a corresponding function, etc.

As an example, if the second interaction mode is the gesture interactionmode, the display of the interface is stopped (e.g., the correspondingwidget is closed, minimized, etc.) if a preset gesture is detected. Insome embodiments, a preset gesture in response to which the display ofthe interface is stopped is different from the function-selectinggesture. For example, the preset gesture in response to which thedisplay of the interface is stopped can be a gesture mapped to switchapplications, to close the interface, etc. Accordingly, confusion withrespect to the gestures that cause the display of the interface to bestopped and the gestures that can invoke a function to be performed isminimized or avoided. Further, the distinction between the gestures canraise operational accuracy.

As an example, if the second interaction mode is a touch interactionmode, then in response to a determination that the hand or input device(e.g., stylus, etc.) exits the field of view of the one or more sensorsof the terminal, or leaves the positions corresponding to positions onthe screen, the display of the interface is stopped (e.g., thecorresponding widget may be closed).

According to various embodiments, a plurality of interaction modes areimplemented in connection with performing a desired function. Forexample, a first interaction mode and a second interaction mode areimplemented in connection with performing a function. The first inputcorresponding to the first interaction mode is used in connection withdisplaying of an interface. The second input corresponding to the secondinteraction mode is used in connection with invoking execution of afunction corresponding to the interface. Various embodiments combine theadvantages of the first interaction mode and the second interaction modeand avoid the defects of implementing only one of the first interactionmode or the second interaction mode to cause the function to beperformed. Various embodiments implement invocation of a functionthrough a first input and a second input and thus may reduce operationaldifficulty for the user while improving operational efficiency for theuser.

According to various embodiments, the interface target position on thescreen matches the position of the hand of the user. Thus, relatednessbetween the user (e.g., the hand of the user) and the interface isincreased, thereby providing an effect according to which the interfaceis provided at a position corresponding to a position of the hand. As aresult, hand operations become more convenient, and the user experienceis improved.

FIG. 3 is a flowchart of a data processing method according to variousembodiments of the present application.

Referring to FIG. 3, process 300 for processing data is provided.Process 300 can implement invocation of a function or control of aterminal. Process 300 can be implemented in connection with process 200of FIG. 2. For example, process 300 can be implemented by process 200 ofFIG. 2. Process 300 can be implemented in connection with interaction400 of FIG. 4, interaction 500 of FIG. 5, interaction 600 of FIG. 6,and/or interaction 700 of FIG. 7. Process 300 can be implemented atleast in part by system 100 of FIG. 1 and/or computer system 1100 ofFIG. 11.

At 310, a voice input is obtained. In some embodiments, 310 isimplemented by 210 of process 200 of FIG. 2. In some embodiments, aterminal receives the first input from a user (e.g., based on a userinteracting with the terminal). The first input is obtained via a firstinteraction mode. The first interaction mode can be configured based onterminal settings, user preferences, historical information pertainingto use of the terminal or interaction with an interface, etc. The firstinteraction mode is a voice interaction mode.

At 320, an interface is provided. In some embodiments, the interface isprovided based at least in part on the voice input. In some embodiments,320 is implemented by 220 of process 200 of FIG. 2. For example, theterminal displays the interface on a display (e.g., a touchscreen).According to various embodiments, the interface is provided based atleast in part on the first input. For example, in response to receivingthe first input, the terminal obtains the interface. The interface canbe preset or preconfigured before the terminal receives the first input,or the interface can be configured or generated based at least in parton the first input. For example, the interface can be configured basedon a particular command associated with the first input, a type of inputcorresponding to the first input, a type of command associated with thefirst input, a first interaction mode via which the first input isreceived, etc.

According to various embodiments, the interface is provided at a targetposition. The target position can be determined based at least in parton the user. For example, the target position matches a position of thehand of the user.

At 330, a gesture input is obtained. In some embodiments, 330 isimplemented by 230 of process 200 of FIG. 2. In some embodiments, aterminal receives the second input from a user (e.g., based on a userinteracting with the terminal). The second input is obtained via asecond interaction mode. The second interaction mode can be configuredbased on terminal settings, user preferences, historical informationpertaining to use of the terminal or interaction with an interface, etc.The second interaction mode can be based at least in part on one or moreof the first input, the first input interaction mode, an applicationcontext, a context of the terminal, etc. The second interaction mode isa gesture interaction mode.

At 340, a function is performed. In some embodiments, the function isperformed based at least in part on the gesture input. The function thatis performed can correspond to a function that selection of which isprovided by the interface. In some embodiments, 340 is implemented by240 of process 200 of FIG. 2. In some embodiments, the function isperformed in response to the second input being obtained (e.g., via thesecond interaction mode). As an example, the function corresponds to thesecond input and/or the second interaction mode. The terminal can storea mapping of inputs to functions, and the terminal can determine thefunction corresponding to the second input based at least in part on themapping of inputs to functions. In some embodiments, the functionperformed in response to the second input is based at least in part onthe second input and one or more of the interface, the first input, thefirst interaction mode, and the second interaction mode.

By combining the naturalness of speech with the conciseness of gestures,various embodiments overcome the problem of lengthy interactions ininstances when only voice interaction modes are implemented, andovercome the problem of difficulties associated with learning andmemorization difficulties in instances when only gesture interactionmodes are implemented. Accordingly, in comparison to the related art,various embodiments are easier for the user to implement functions. Thevoice input can be integrated with remote gesture input (e.g., both thevoice interaction mode and the gesture interaction mode can beimplemented in connection with performing a desired function), whereinan interface is called up by one or more voice inputs. In someembodiments, the interface is called up in response to a single voiceinput. Accordingly, a number of voice interactions to navigate anapplication or a menu of a terminal to invoke an interface is reduced.

According to various embodiments, gesture inputs can be quickly andeasily generated in connection with the gesture interaction mode inorder to implement (e.g., invoke execution of) the function. Differenttypes of functions displayed in different interface can share samegesture. Accordingly, a user has less difficulty in learning andmemorizing gestures. As an example, different types of functions such asvolume, brightness, saturation, contrast, temperature, and wind speeddisplayed in different interfaces can has the same gesture. In someembodiments, gestures mapped to a corresponding function are furtherbased on the first input.

In some embodiments, the target position at which the interface isprovided on the screen is based at least in part on the user. Forexample, the target position matches the position of the hand of theuser (e.g., a position of the hand in relation to one or more sensors orin relation to the terminal). Thus, relatedness between the user (e.g.,the hand of the user) and the interface is increased, thereby enablingthe user to experience an effect in which the interface is positioned inaccordance with a position of the hand. As a result, hand operationsbecome more convenient, and the user experience is improved.

FIG. 4 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

Referring to FIG. 4, interaction 400 between a user and a terminal isprovided. Interaction 400 can be implemented in connection withinvocation of a function or control of a terminal. Interaction 400 canbe implemented in connection with process 200 of FIG. 2 and/or process300 of FIG. 3. For example, interaction 400 can be implemented byprocess 200 of FIG. 2 and/or process 300 of FIG. 3. Interaction 400 canbe implemented at least in part by system 100 of FIG. 1 and/or computersystem 1100 of FIG. 11.

As illustrated in FIG. 4, a flow diagram of an interaction 400 between auser and a terminal is provided. At 410, the user inputs a first input(e.g., a voice input). If a voice input (e.g., “volume”) is receivedfrom a user, the target function is determined. For example, if thevoice input corresponds to “volume” or “change the volume,” the targetfunction can be determined to be a “volume function,” etc. The targetfunction is determined so as to correspond to the voice input. In someembodiments, the target function is determined based on a query of oneor more keywords comprised in the voice input performed with respect toa mapping of keywords to functions. In addition, at 420, a hand hoversover the front of the screen. In response to a determination that a handis detected hovering in front of the screen, the target position isdetermined according to the hand position. Interface 401 is configuredbased at least in part on the voice input and/or the position of thehand that is detected to be hovering over the screen. At 430, interface401 is displayed at the target position on the screen. Interface 401 caninclude: a widget and an audio icon. The widget includes a controlelement 403 and a guide container 402, and the control element isconfigured to slide within the guide container. The user can input agesture input or a touch input to slide the control element within theguide container to invoke a control function (e.g., to control thevolume). According to interaction 400, the hand of the user is detectedto be hovering over a bottom left area of the screen at 420, and as aresult, interface 401 is provided at a bottom left corner of the screenat 430.

FIG. 5 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

Referring to FIG. 5, interaction 500 between a user and a terminal isprovided. Interaction 500 can be implemented in connection withinvocation of a function or control of a terminal. Interaction 500 canbe implemented in connection with process 200 of FIG. 2 and/or process300 of FIG. 3. For example, interaction 500 can be implemented byprocess 200 of FIG. 2 and/or process 300 of FIG. 3. Interaction 500 canbe implemented at least in part by system 100 of FIG. 1 and/or computersystem 1100 of FIG. 11.

As illustrated, a flow diagram of an interaction 500 between a user anda terminal is provided. At 510, the user inputs a first input (e.g., avoice input). If a voice input (e.g., “volume”) is received from a user,the target function is determined. For example, if the voice inputcorresponds to “volume” or “change the volume,” the target function canbe determined to be a “volume function,” etc. The target function isdetermined so as to correspond to the voice input. In some embodiments,the target function is determined based on a query of one or morekeywords comprised in the voice input performed with respect to amapping of keywords to functions. In addition, at 520, a hand hoversover the front of the screen. In response to a determination that a handis detected hovering in front of the screen, the target position isdetermined according to the hand position. Interface 501 is configuredbased at least in part on the voice input and/or the position of thehand that is detected to be hovering over the screen. At 530, interface501 is displayed at the target position on the screen. Interface 501includes a widget and an audio icon. The widget includes a controlelement 503 and a guide container 502, and the control element isconfigured to slide within the guide container. The user can input agesture input or a touch input to slide the control element within theguide container to invoke a control function (e.g., to control thevolume). According to interaction 500, the hand of the user is detectedto be hovering over a bottom right area of the screen at 520, and as aresult, interface 501 is provided at a bottom right corner of the screenat 530.

A difference between interaction 500 and interaction 400 of FIG. 4includes the target position of interface 501 being different from thetarget position of interface 401. Specifically, the target position ofinterface 401 is lower-left, and the target position of interface 501 islower-right. According to various embodiments, the target positions aredetermined based at least in part on the position of the hand. In someembodiments, the target positions are determined based at least in parton the projection positions of the hand with respect to the screen.According to various embodiments, the position of hand is detected bycamera or some other sensors, and the projection of hand on the screenis the projection position.

FIG. 6 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

Referring to FIG. 6, interaction 600 between a user and a terminal isprovided. Interaction 600 can be implemented in connection withinvocation of a function or control of a terminal. Interaction 600 canbe implemented in connection with process 200 of FIG. 2 and/or process300 of FIG. 3. For example, interaction 600 can be implemented byprocess 200 of FIG. 2 and/or process 300 of FIG. 3. Interaction 600 canbe implemented at least in part by system 100 of FIG. 1 and/or computersystem 1100 of FIG. 11.

As illustrated, a flow diagram of an interaction 600 between a user anda terminal is provided. At 610, the user inputs a first input (e.g., avoice input). If a voice input (e.g., “volume”) is received from theuser while a page (e.g., a user interface screen) is being provided bythe terminal, the voice command is determined to be a target function.For example, in response to receiving the voice input while the terminalprovides any page (e.g., a navigation page, a music page, a mediaplayback page, an application page, a gaming page, etc.), the terminaldetermines that the target function corresponds to the voice input. Forexample, if the voice input corresponds to “volume” or “change thevolume,” the target function can be determined to be a “volume”function, etc. The target function is determined so as to correspond tothe voice input. In addition, the target position is determined based atleast in part on a position of the user (e.g., a position of the hand ofthe user). In response to determining the target function and a positionof the user (e.g., a target position of the interface), the terminalinvokes interface 601 and provides interface 601 at the target positionon the screen. Interface 601 can include a widget and an audio icon. Thewidget can include a control element and a guide container. The widgetis configured such that the control element can slide within the guidecontainer. According to various embodiments, the widget is configured toallow the control element to slide along a path (e.g., a horizontalline, a vertical line, etc.). Interaction 600 illustrates the widgetconfigured to allow for the control element to be adjusted (e.g., slide)in a horizontal path. If a rightward slide operation is received (e.g.,from the user), the control piece may be slid rightward. In someembodiments, the widget and/or interface is used to implement the volumeadjustment function (e.g., via control of the control element).

The rightward slide operation described above can be implemented in thegesture interaction mode or the touch interaction mode. In someembodiments, in response to interface 601 being displayed, the terminaldetermines whether a gesture input or touch input is received and thatsuch gesture input or touch input satisfies one or more criteria (e.g.,a gesture input corresponding to a downward swipe, etc.). For example,one or more predefined inputs can be mapped to functions different fromthe target function, such as exiting out of control of the targetfunction, etc. At 650, if the terminal determines that the gesture inputor touch input is received and that such gesture input or touch inputsatisfies one or more criteria corresponding to exiting control of thetarget function, etc., then display of interface 601 is stopped. Forexample, at 650, if a downward movement gesture is detected, interface601 is closed.

At 620, the terminal determines that a gesture input or touch input isreceived, and that such gesture input or touch input corresponds to aninput to control a slide of the control element to the right. Forexample, the touch input is a touch and drag of the control element inthe rightwards direction. As another example, a gesture input is agesture of the hand of the user moving from a left position to arightwards position. At 630, in response to determining that a gestureinput or touch input is received, and that such gesture input or touchinput corresponds to an input to control a slide of the control elementto the right, the corresponding function is provided (e.g., the volumeis controlled such as to increase the volume).

At 630, while interface 601 is being displayed, the terminal determineswhether a gesture input or touch input is received and that such gestureinput or touch input satisfies one or more criteria (e.g., a gestureinput corresponding to a downward swipe, etc.). For example, one or morepredefined inputs can be mapped to functions different from the targetfunction, such as exiting out of control of the target function, etc. At640, if the terminal determines that the gesture input or touch input isreceived and that such gesture input or touch input satisfies one ormore criteria corresponding to exiting control of the target function,etc., then display of interface 601 is stopped. For example, at 640, ifa downward movement gesture is detected, interface 601 is closed.

Please note that the rightward slide operation described above is merelyan example of a gesture input. Various gestures can also include:vertical shift operations, circling operations, rotating operations, aswell as other operations.

FIG. 7 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

Referring to FIG. 7, interaction 700 between a user and a terminal isprovided. Interaction 700 can be implemented in connection withinvocation of a function or control of a terminal. Interaction 700 canbe implemented in connection with process 200 of FIG. 2 and/or process300 of FIG. 3. For example, interaction 700 can be implemented byprocess 200 of FIG. 2 and/or process 300 of FIG. 3. Interaction 700 canbe implemented at least in part by system 100 of FIG. 1, and/or computersystem 1100 of FIG. 11.

As illustrated, a flow diagram of an interaction 700 between a user anda terminal is provided. At 710, the user inputs a first input (e.g., avoice input). If a voice input (e.g., “fan speed”) is received from theuser while a page (e.g., any page) is being provided by the terminal,the voice command is determined to be a target function. For example, inresponse to receiving the voice input while the terminal provides anypage (.g., a navigation page, a music page, a media playback page, anapplication page, a gaming page, etc.), the terminal determines that thetarget function corresponds to the voice input. For example, if thevoice input corresponds to “fan speed” or “change the fan,” the targetfunction can be determined to be a “fan speed,” etc. The target functionis determined so as to correspond to the voice input. In addition, thetarget position is determined based at least in part on a position ofthe user (e.g., a position of the hand of the user). In response todetermining the target function and a position of the user (e.g., aposition of the user's hand, which corresponds to a target position ofthe interface), the terminal invokes interface 701 and providesinterface 701 at the target position on the screen. Interface 701 caninclude: a widget and a fan speed icon. The widget can include: acontrol element and a guide container. The widget can be configured suchthat the control element can slide within the guide container. Accordingto various embodiments, the widget is configured to allow the controlelement to slide along a path (e.g., a horizontal line, a vertical line,etc.). Interaction 700 illustrates the widget configured to allow forthe control element to be adjusted (e.g., slide) in a horizontal path.If a rightward slide operation is received (e.g., from the user), thecontrol piece may be slid rightward. In some embodiments, the widgetand/or interface is used to implement the fan speed function (e.g., viacontrol of the control element).

The rightward slide operation described above can be implemented in thegesture interaction mode or the touch interaction mode. In someembodiments, in response to interface 701 being displayed, the terminaldetermines whether a gesture input or touch input is received and thatsuch gesture input or touch input satisfies one or more criteria (e.g.,a gesture input corresponding to a downward swipe, etc.). For example,one or more predefined inputs can be mapped to functions different fromthe target function, such as exiting out of control of the targetfunction, etc. At 750, if the terminal determines that the gesture inputor touch input is received and that such gesture input or touch inputsatisfies one or more criteria corresponding to exiting control of thetarget function, etc., then display of interface 701 is stopped. Forexample, at 750, if a downward movement gesture is detected, interface701 is closed.

At 720, the terminal determines that a gesture input or touch input isreceived, and that such gesture input or touch input corresponds to aninput to control a slide of the control element to the right. Forexample, the touch input is a touch and drag in the rightwardsdirection. As another example, a gesture input is a gesture of the handof the user moving from a left position to a rightwards position. At730, in response to determining that a gesture input or touch input isreceived, and that such gesture input or touch input corresponds to aninput to control a slide of the control element to the right, thecorresponding function is provided (e.g., the fan speed is controlledsuch as to increase the fan speed).

At 730, while interface 701 is being displayed, the terminal determineswhether a gesture input or touch input is received and that such gestureinput or touch input satisfies one or more criteria (e.g., a gestureinput corresponding to a downward swipe, etc.). For example, one or morepredefined inputs can be mapped to functions different from the targetfunction, such as exiting out of control of the target function, etc. At740, if the terminal determines that the gesture input or touch input isreceived and that such gesture input or touch input satisfies one ormore criteria corresponding to exiting control of the target function,then display of interface 701 is stopped. For example, at 740, if adownward movement gesture is detected, interface 701 is closed.

Please note that the rightward slide operation described above is merelyan example of a gesture input. Various gestures can include: verticalshift operations, circling operations, rotating operations, or otheroperations.

Various embodiments make integrated use of voice input and remotegesture input, such that an interface is invoked in response to a singlevoice input. Accordingly, a number of voice interactions can be reduced.

Various embodiments implement the function corresponding to theinterface quickly and easily via a gesture input that is input in thegesture interaction mode. The displayed interface can cause differenttypes of functions to be mapped to a same gesture. For example, amapping of gestures to types of functions or of gestures to functionscan be a one-to-many mapping. As a result of fewer gesture patternsbeing used to implement a particular number of functions as compared tothe related art, users experience less difficulty in learning andmemorizing gestures.

According to various embodiments, the target position at which theinterface is provided on the screen matches a position of the user(e.g., a position of the user with respect to the terminal). Forexample, the target position at which the interface is provided on thescreen matches a position of the hand of the user. Thus, relatednessbetween the user (e.g., the hand of the user) and the interface isincreased, and an effect in which the interface is provided tocorrespond to a location of the hand of the user is implemented. As aresult, hand operations become more convenient, and the user experienceis improved.

FIG. 8 is a flowchart of a data processing method according to variousembodiments of the present application.

Referring to FIG. 8, process 800 for processing data is provided.Process 800 can implement invocation of a function or control of aterminal. Process 800 can be implemented in connection with process 200of FIG. 2. For example, process 800 can be implemented by process 200 ofFIG. 2. Process 800 can be implemented in connection with interaction900 of FIG. 9 and/or interaction 1000 of FIG. 10. Process 800 can beimplemented at least in part by system 100 of FIG. 1 and/or computersystem 1100 of FIG. 11.

At 810, a voice input is obtained. In some embodiments, 810 isimplemented by 210 of process 200 of FIG. 2. In some embodiments, aterminal receives the first input from a user (e.g., based on a userinteracting with the terminal). The first input is obtained via a firstinteraction mode. The first interaction mode can be configured based onterminal settings, user preferences, historical information pertainingto use of the terminal or interaction with an interface, etc. The firstinteraction mode is a voice interaction mode.

At 820, an interface is provided. In some embodiments, the interface isprovided based at least in part on the voice input. In some embodiments,820 is implemented by 220 of process 200 of FIG. 2. For example, theterminal displays the interface on a display (e.g., a touchscreen).According to various embodiments, the interface is provided based atleast in part on the first input. For example, in response to receivingthe first input, the terminal obtains the interface. The interface canbe preset or preconfigured before the terminal receives the first input,or the interface can be configured or generated based at least in parton the first input. For example, the interface can be configured basedon a particular command associated with the first input, a type of inputcorresponding to the first input, a type of command associated with thefirst input, a first interaction mode via which the first input isreceived, etc.

According to various embodiments, the interface is provided at a targetposition. The target position can be determined based at least in parton the user. For example, the target position matches a position of thehand of the user.

At 830, a touch input is obtained. In some embodiments, 830 isimplemented by 230 of process 200 of FIG. 2. In some embodiments, aterminal receives the second input from a user (e.g., based on a userinteracting with the terminal). The second input is obtained via asecond interaction mode. The second interaction mode can be configuredbased on terminal settings, user preferences, historical informationpertaining to use of the terminal or interaction with an interface, etc.The second interaction mode can be based at least in part on one or moreof the first input, the first input interaction mode, an applicationcontext, a context of the terminal, etc. The second interaction mode isa touch interaction mode.

The touch input can be a tap, a touch and hold, a swipe, etc. Varioustouch inputs can be implemented in connection with invoking execution ofa corresponding function.

At 840, a function is performed. In some embodiments, the function isperformed based at least in part on the touch input. The function thatis performed can correspond to a function that selection of which isprovided by the interface. In some embodiments, 840 is implemented by240 of process 200 of FIG. 2. In some embodiments, the function isperformed in response to the second input being obtained (e.g., via thesecond interaction mode). As an example, the function corresponds to thesecond input and/or the second interaction mode. The terminal can storea mapping of inputs to functions, and the terminal can determine thefunction corresponding to the second input based at least in part on themapping of inputs to functions. In some embodiments, the functionperformed in response to the second input is based at least in part onthe second input and one or more of the interface, the first input, thefirst interaction mode, and the second interaction mode.

Various embodiments implement a corresponding function based at least inpart on a combination of a voice input and a touch input. By combiningthe naturalness of speech with the conciseness of touch, variousembodiments overcome the problem of the related art according to whichinteractions are lengthy because the user-terminal interactions areperformed using only voice interaction modes (e.g., without theintegration of a plurality of interaction modes), and the problem oflong operational paths associated with user-terminal interactions areperformed using only a touch interaction mode. Accordingly, variousembodiments facilitate the invocation of functions by a user withrelative ease as compared to the related art.

FIG. 9 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

Referring to FIG. 9, interaction 900 between a user and a terminal isprovided. Interaction 900 can be implemented in connection withinvocation of a function or control of a terminal. Interaction 900 canbe implemented in connection with process 200 of FIG. 2 and/or process800 of FIG. 8. For example, interaction 900 can be implemented byprocess 200 of FIG. 2 and/or process 800 of FIG. 8. Interaction 900 canbe implemented at least in part by system 100 of FIG. 1 and/or computersystem 1100 of FIG. 11.

As illustrated in FIG. 9, a flow diagram of an interaction 900 between auser and a terminal is provided. At 910, the user inputs a first input(e.g., a voice input). If a voice input (e.g., “volume”) is receivedfrom a user, the target function is determined. For example, if thevoice input corresponds to “volume” or “change the volume,” the targetfunction can be determined to be a “volume function,” etc. The targetfunction is determined so as to correspond to the voice input. In someembodiments, the target function is determined based on a query of oneor more keywords comprised in the voice input performed with respect toa mapping of keywords to functions. In addition, at 920, a touch inputis made to the terminal. For example, the touch input is input via ahand of a user tapping on a screen. The terminal can detect a touchinput in response to a touchscreen of the terminal being touched. Inresponse to a determination that a touch input is made to thetouchscreen (e.g., the hand of the user taps the touchscreen), thetarget position is determined according to the hand position (e.g.,according to a position at which the touchscreen is tapped). Interface901 is configured based at least in part on the voice input and/or theposition of the hand (e.g., the location of the touchscreen at which thetouch input is input). At 930, interface 901 is displayed at the targetposition on the screen. Interface 901 can include: a widget and an audioicon. The widget can include: a control element and a guide container,and the control element can slide within the guide container. The usercan input a gesture input or a touch input to slide the control elementwithin the guide container to invoke a control function (e.g., tocontrol the volume). According to interaction 900, the hand of the useris detected to be over a bottom left area of the screen at 920 (e.g.,the touch input to the touchscreen is input at an area corresponding tothe bottom left area of the screen), and as a result, interface 901 isprovided at a bottom left corner of the screen at 930.

FIG. 10 is a diagram of an interaction between a user and a terminalaccording to various embodiments of the present application.

Referring to FIG. 10, interaction 1000 between a user and a terminal isprovided. Interaction 1000 can be implemented in connection withinvocation of a function or control of a terminal. Interaction 1000 canbe implemented in connection with process 200 of FIG. 2 and/or process800 of FIG. 8. For example, interaction 1000 can be implemented byprocess 200 of FIG. 2 and/or process 800 of FIG. 8. Interaction 1000 canbe implemented at least in part by system 100 of FIG. 1 and/or computersystem 1100 of FIG. 11.

As illustrated in FIG. 10, a flow diagram of an interaction 1000 betweena user and a terminal is provided. At 1010, the user inputs a firstinput (e.g., a voice input). If a voice input (e.g., “volume”) isreceived from a user, the target function is determined. For example, ifthe voice input corresponds to “volume” or “change the volume,” thetarget function can be determined to be a “volume function,” etc. Thetarget function is determined so as to correspond to the voice input. Insome embodiments, the target function is determined based on a query ofone or more keywords comprised in the voice input performed with respectto a mapping of keywords to functions. In addition, at 1020, a touchinput is made to the terminal. For example, the touch input is input viaa hand of a user tapping on a screen. The terminal can detect a touchinput in response to a touchscreen of the terminal being touched. Inresponse to a determination that a touch input is made to thetouchscreen (e.g., the hand of the user taps the touchscreen), thetarget position is determined according to the hand position (e.g.,according to a position at which the touchscreen is tapped). Interface1001 is configured based at least in part on the voice input and/or theposition of the hand (e.g., the location of the touchscreen at which thetouch input is input). At 1030, interface 1001 is displayed at thetarget position on the screen. Interface 1001 can include: a widget andan audio icon. The widget can include: a control element and a guidecontainer, and the control element can slide within the guide container.The user can input a gesture input or a touch input to slide the controlelement within the guide container to invoke a control function (e.g.,to control the volume). According to interaction 1000, the hand of theuser is detected to be over a top right area of the screen at 1020(e.g., the touch input to the touchscreen is input at an areacorresponding to the top right area of the screen), and as a result,interface 1001 is provided at a top right corner of the screen at 1030.

A difference between interaction 1000 and interaction 900 of FIG. 9 isthe target position of an interface 1001 is different from the targetposition of the interface 901. Specifically, the target position of theinterface 901 is lower-left, and the target position of interface 1001is upper-right. The target positions can be obtained according to handposition. In some embodiments, the position of the control element maybe the position at which the touch input is input to the touchscreen(e.g., the location at which the hand touches the touchscreen).

Various embodiments implement a corresponding function based at least inpart on a combination of a voice input and a touch input. By combiningthe naturalness of speech with the conciseness of touch, variousembodiments overcome the problem of the related art according to whichinteractions are lengthy because the user-terminal interactions areperformed using only voice interaction modes (e.g., without theintegration of a plurality of interaction modes), and the problem oflong operational paths associated with user-terminal interactions areperformed using only a touch interaction mode. Accordingly, variousembodiments facilitate the invocation of functions by a user withrelative ease as compared to the related art.

First, an interface is invoked and/or displayed in response to a singlevoice input. Accordingly, reducing the number of voice interactions usedin connection with invoking a function is reduced.

Moreover, an interface can quickly and directly be obtained via a singleinput, which decreases the operating paths involved in implementing afunction through the touch interaction mode. Accordingly, the relativeoperational difficulty for the user is decreased.

According to various embodiments, the target position at which theinterface is provided on the screen matches a position of the user(e.g., a position of the user with respect to the terminal). Forexample, the target position at which the interface is provided on thescreen matches a position of the hand of the user. Thus, relatednessbetween the user (e.g., the hand of the user) and the interface isincreased, and an effect in which the interface is provided tocorrespond to a location of the hand of the user is implemented. As aresult, hand operations become more convenient, and the user experienceis improved.

FIG. 11 is a functional diagram of a computer system for data processingaccording to various embodiments of the present application.

Referring to FIG. 11, computer system 1100 for invoking a function orcontrolling a terminal is provided. Computer system 1100 can implementat least part of process 200 of FIG. 2, process 300 of FIG. 3,interaction 400 of FIG. 4, interaction 500 of FIG. 5, interaction 600 ofFIG. 6, interaction 700 of FIG. 7, process 800 of FIG. 8, interaction900 of FIG. 9, and/or interaction 1000 of FIG. 10. Computer system 1100can be implemented at least in part of system 100 of FIG. 1.

Processor 1102 is coupled bi-directionally with memory 1110, which caninclude a first primary storage, typically a random access memory (RAM),and a second primary storage area, typically a read-only memory (ROM).As is well known in the art, primary storage can be used as a generalstorage area and as scratch-pad memory, and can also be used to storeinput data and processed data. Primary storage can also storeprogramming instructions and data, in the form of data objects and textobjects, in addition to other data and instructions for processesoperating on processor 1102. Also as is well known in the art, primarystorage typically includes basic operating instructions, program code,data, and objects used by the processor 1102 to perform its functions(e.g., programmed instructions). For example, memory 1110 can includeany suitable computer-readable storage media, described below, dependingon whether, for example, data access needs to be bi-directional oruni-directional. For example, processor 1102 can also directly and veryrapidly retrieve and store frequently needed data in a cache memory (notshown). The memory can be a non-transitory computer-readable storagemedium.

A removable mass storage device 1112 provides additional data storagecapacity for the computer system 1100, and is coupled eitherbi-directionally (read/write) or uni-directionally (read only) toprocessor 1102. For example, storage 1112 can also includecomputer-readable media such as magnetic tape, flash memory, PC-CARDS,portable mass storage devices, holographic storage devices, and otherstorage devices. A fixed mass storage 1120 can also, for example,provide additional data storage capacity. The most common example ofmass storage 1120 is a hard disk drive. Mass storage device 1112 andfixed mass storage 1120 generally store additional programminginstructions, data, and the like that typically are not in active use bythe processor 1102. It will be appreciated that the information retainedwithin mass storage device 1112 and fixed mass storage 1120 can beincorporated, if needed, in standard fashion as part of memory 1110(e.g., RAM) as virtual memory.

In addition to providing processor 1102 access to storage subsystems,bus 1114 can also be used to provide access to other subsystems anddevices. As shown, these can include a display monitor 1118, a networkinterface 1116, a keyboard 1104, and a pointing device 1106, as well asan auxiliary input/output device interface, a sound card, speakers, andother subsystems as needed. For example, the pointing device 1106 can bea mouse, stylus, track ball, or tablet, and is useful for interactingwith a graphical user interface.

The network interface 1116 allows processor 1102 to be coupled toanother computer, computer network, or telecommunications network usinga network connection as shown. For example, through the networkinterface 1116, the processor 1102 can receive information (e.g., dataobjects or program instructions) from another network or outputinformation to another network in the course of performingmethod/process steps. Information, often represented as a sequence ofinstructions to be executed on a processor, can be received from andoutputted to another network. An interface card or similar device andappropriate software implemented by (e.g., executed/performed on)processor 1102 can be used to connect the computer system 1100 to anexternal network and transfer data according to standard protocols. Forexample, various process embodiments disclosed herein can be executed onprocessor 1102, or can be performed across a network such as theInternet, intranet networks, or local area networks, in conjunction witha remote processor that shares a portion of the processing. Additionalmass storage devices (not shown) can also be connected to processor 1102through network interface 1116.

An auxiliary I/O device interface (not shown) can be used in conjunctionwith computer system 1100. The auxiliary I/O device interface caninclude general and customized interfaces that allow the processor 1102to send and, more typically, receive data from other devices such asmicrophones, touch-sensitive displays, transducer card readers, tapereaders, voice or handwriting recognizers, biometrics readers, cameras,portable mass storage devices, and other computers.

The computer system shown in FIG. 11 is but an example of a computersystem suitable for use with the various embodiments disclosed herein.Other computer systems suitable for such use can include additional orfewer subsystems. In addition, bus 1114 is illustrative of anyinterconnection scheme serving to link the subsystems. Other computerarchitectures having different configurations of subsystems can also beutilized.

The systems, means, modules, or units illustrated by the aboveembodiments specifically may be implemented by computer chips orentities or by products having certain functions. A typical implementingdevice is a computer. The particular form a computer may take may be apersonal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device,email receiving device, game console, tablet computer, wearable device,or a combination of any of these devices.

In a typical configuration, a computer comprises one or more processors(CPUs), input/output ports, network interfaces, and memory.

Memory may include the following forms in computer-readable media:volatile memory, random access memory (RAM), and/or non-volatile memory,e.g., read-only memory (ROM) or flash RAM. Memory is an example of acomputer-readable medium.

Accordingly, various embodiments further provide a computer-readablemedium storing computer programs. When executed, the computer programsare capable of implementing all the steps executable by a server in themethod embodiments described above.

The memory in FIG. 11 may take the form of any type of volatile ornon-volatile storage device or combination thereof. Examples includestatic random access memory (SRAM), electrically erasable programmableread-only memory (EEPROM), erasable programmable read-only memory(EPROM), programmable read-only memory (PROM), read-only memory (ROM),magnetic memory, flash memory, magnetic disks, and optical disks.

Computer system 1100 of FIG. 11 can include a communication componentthat is configured to facilitate wired or wireless communication betweenthe device on which the communication component is located and otherdevices. The device where the communication component is located mayaccess wireless networks based on a communications standard such asWi-Fi, 2G, 3G, or a combination thereof. In an exemplary embodiment, thecommunication component receives via broadcast channels broadcastsignals or broadcast-related information from external broadcastmanagement systems. In an exemplary embodiment, said communicationcomponent further comprises a near-field communication module forpromoting short-range communication. For example, it can be achieved inthe NFC module on the basis of radio-frequency identification (RFID)technology, Infrared Data Association (IrDA) technology, ultra-wide band(UWB) technology, Bluetooth (BT) technology, and other technology.

Display 1118 in FIG. 11 above includes a screen, and the screen mayinclude a liquid crystal display (LCD) or a touch panel (TP). If thescreen comprises a touch panel, the screen may be implemented as atouchscreen to receive input signals from the user. The touch panelcomprises one or more touch sensors to detect touch, sliding actions,and gestures on the touch panel. Said touch sensor can not only detectthe boundaries of touch or slide actions, but also measure the durationand pressure related to said touch or slide operations.

Computer system 1100 of FIG. 11 can include a power supply componentthat is configured to provide electric power to all components in thedevice where the power supply component is located. The power supplycomponent may include a power supply management system, one or morepower supplies, and other components related to generating, managing,and allocating power to the device where the power supply component islocated.

A person skilled in the art should understand that the embodiments ofthe present invention can be provided as methods, systems or computerprogram products. Therefore, the present invention may take the form ofcomplete hardware embodiments, complete software embodiments, orembodiments that combine software and hardware. Moreover, the presentinvention may take the form of computer program products implemented onone or more computer-operable storage media (including but not limitedto magnetic disk storage, CD-ROMs, and optical storage) containingcomputer-operable program code.

Please note that all the method embodiments have been presented as aseries of a combination of actions in order to simplify the description.However, persons skilled in the art should know that embodiments of thepresent application are not limited by the action sequences that aredescribed, for some of the steps may make use of another sequence or beimplemented simultaneously in accordance with embodiments of the presentapplication. Secondly, persons skilled in the art should also know thatthe embodiments described in the specification are all preferredembodiments. The actions that they involve are not necessarily requiredby embodiments of the present application. With regard to an embodiment,at least one of the processors 1102 may be packaged together with thelogic of one or more controllers (e.g., memory controllers) of a systemcontrol module. With regard to an embodiment, at least one of theprocessors 1102 may be packaged together with the logic of one or morecontrollers of the system control logic module to form asystem-in-package (SiP). With regard to an embodiment, at least one ofthe processors 1102 may be integrated with the logic of one or morecontrollers of the system control module in a single new product. Withregard to an embodiment, at least one of the processors 1102 may beintegrated with the logic of one or more controllers of the systemcontrol module to form a system-on-a-chip (SoC) on a single chip.

If the display comprises a touch panel, the display screen can beimplemented as a touchscreen display to receive input signals from theuser. The touch panel can include one or more touch sensors to detecttouch, sliding actions, and gestures on the touch panel. The touchsensor can not only detect the boundaries of touch or slide actions, butalso measure the duration and pressure related to said touch or slideoperations.

Various embodiments further provide a non-volatile, readable storagemedium. The storage medium stores one or more modules (programs), andwhen these one or more modules are applied on an apparatus, the modulescan cause the apparatus to execute the instructions of each method inembodiments of the present application.

With regard to the apparatus in the embodiment described above, theparticular ways in which the various modules execute operations havealready been described in detail in the method embodiments and will notbe explicated in detail here. One may refer to some of the explanationsin the method embodiments where appropriate.

Each of the embodiments contained in this specification is described ina progressive manner. The explanation of each embodiment focuses onareas of difference from the other embodiments, and the descriptionsthereof may be mutually referenced regarding portions of each embodimentthat are identical or similar.

Embodiments of the present application are described with reference toflow charts and/or block diagrams based on methods, devices (systems),and computer program products of the embodiments of the presentapplication. Please note that each process and/or block within theflowcharts and/or block diagrams and combinations of processes and/orblocks within the flowcharts and/or block diagrams can be implemented bycomputer instructions. These computer program instructions can beprovided to a general-purpose computer, a special-purpose computer,embedded processors, or processors of other programmable data-processingapparatuses to give rise to a machine such that the instructionsexecuted by the processors of the computer or other programmabledata-processing apparatuses give rise to means used to implement thefunctions specified in one or more processes in a flowchart and/or inone or more blocks in a block diagram.

These computer program instructions can also be stored incomputer-readable memory that can guide computers or other programmabledata-processing apparatuses to operate according to specific modes, withthe result that the instructions stored in this computer-readable memorygive rise to products that include command means. These command meansimplement the functions specified in one or more processes in a flowchart and/or one or more blocks in a block diagram.

These computer program instructions can also be loaded onto a computeror other programmable data-processing apparatus, with the result that aseries of operating steps are executed on a computer or otherprogrammable apparatus so as to give rise to computer processing. Inthis way, the instructions executed on a computer or other programmableapparatus provide steps for implementing the functions specified by oneor more processes in a flow chart and/or one or more blocks in a blockdiagram.

Although preferred embodiments of the present application have alreadybeen described, persons skilled in the art can make other modificationsor revisions to these embodiments once they grasp the basic creativeconcept. Therefore, the attached claims are to be interpreted asincluding the preferred embodiments as well as all modifications andrevisions falling within the scope of the embodiments of the presentapplication.

Lastly, it must also be explained that, in this document, relationalterms such as “first” or “second” are used only to differentiate betweenone entity or operation and another entity or operation, withoutnecessitating or implying that there is any such actual relationship orsequence between these entities or operations. Furthermore, the terms“comprise” or “contain” or any of their variants are to be taken intheir non-exclusive sense. Thus, processes, methods, objects, or meansthat comprise a series of elements not only comprise those elements, butalso comprise other elements that have not been explicitly listed orelements that are intrinsic to such processes, methods, objects, ormeans. In the absence of further limitations, for an element that islimited by the phrase “comprises a(n) . . . ”, the existence ofadditional identical elements in processes, methods, objects, or meansthat comprise said elements is not excluded.

Detailed introductions were provided above to a data processing method,a data processing means, an apparatus, and a machine-readable mediumprovided by the present application. This document has applied specificexamples to explain the principles and implementations of the presentapplication. The above descriptions of the embodiments are only for thepurpose of aiding the understanding of the methods and core concepts ofthe present application. A person with ordinary skill in the art willalways be able to make modifications in keeping with the idea of thepresent application to specific embodiments and scopes of theapplication. The content of this specification should not be understoodas limiting the present application.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A method, comprising: obtaining, by one or moreprocessors, a first input, the first input being input via a firstinteraction mode; providing, by the one or more processors, aninterface, the interface being provided based at least in part on thefirst input; obtaining, by the one or more processors, a second input,the second input being input via a second interaction mode, the secondinteraction mode being different from the first interaction mode; andperforming, by the one or more processors, a function corresponding tothe interface, the function being based at least in part on one or moreof the first input, the first interaction mode, the second input, andthe second interaction mode.
 2. The method of claim 1, wherein theinterface is provided at a target position, and the target position isdetermined based at least in part on a position of a user in relation toa terminal when the first input is input to the terminal.
 3. The methodof claim 1, wherein the first interaction mode is a voice interactionmode, and the second interaction mode is a gesture interaction mode or atouch interaction mode; or the first interaction mode is the touchinteraction mode, and the second interaction mode is the gestureinteraction mode or the voice interaction mode; or the first interactionmode is the gesture interaction mode, and the second interaction mode isthe touch interaction mode or the voice interaction mode.
 4. The methodof claim 1, wherein the first interaction mode is a voice interactionmode, the first input is a voice command, and the voice commandindicates the function to be invoked in connection with the secondinput.
 5. The method of claim 4, wherein the second interaction mode isdetermined to be a touch interaction mode in response to a determinationthat a touch input is input to a touchscreen of a terminal.
 6. Themethod of claim 4, wherein the second interaction mode is determined tobe a gesture interaction mode in response to a determination that agesture is input before a touch input is input to a touchscreen of aterminal.
 7. The method of claim 1, wherein the interface is provided ata target position, the target s position is determined based at least inpart on a position of a user in relation to a terminal at a time atwhich the first input is input to the terminal, and the position of theuser in relation to the terminal corresponds to a position of a hand ofthe user.
 8. The method of claim 1, wherein the interface comprises: awidget, the widget including a control element, and the control elementis configured to respond to a slide event.
 9. The method of claim 8,wherein a position of the control element matches a position of a handof a user.
 10. The method of claim 1, wherein the interface comprises:function information.
 11. The method of claim 10, wherein the secondinteraction mode is a gesture interaction mode, and the performing thefunction corresponding to the interface comprises: in response to adetermination that the second input corresponds to a function-selectinggesture, executing a target function corresponding to thefunction-selecting gesture.
 12. The method of claim 11, wherein aposition at which the function information is displayed is mapped to anorientation of the function-selecting gesture.
 13. The method of claim1, further comprising: stopping the display of the interface in responseto a determination that a third input satisfies one or more predefinedconditions.
 14. A device, comprising: one or more processors configuredto: obtain a first input, the first input being input via a firstinteraction mode; provide an interface, the interface being displayedbased at least in part on the first input; obtain a second input, thesecond input being input via a second interaction mode, the secondinteraction mode being different from the first interaction mode; andperform a function corresponding to the interface, the function beingbased at least in part on one or more of the first input, the firstinteraction mode, the second input, and the second interaction mode; andone or more memories coupled to the one or more processors, configuredto provide the one or more processors with instructions.
 15. The deviceof claim 14, wherein the interface is provided at a target position, andthe target position is determined based at least in part on a positionof a user in relation to the device at a time at which the first inputis input to the device.
 16. The device of claim 14, wherein the firstinteraction mode is a voice interaction mode, and the second interactionmode is a gesture interaction mode or a touch interaction mode; or thefirst interaction mode is the touch interaction mode, and the secondinteraction mode is the gesture interaction mode or the voiceinteraction mode; or the first interaction mode is the gestureinteraction mode, and the second interaction mode is the touchinteraction mode or the voice interaction mode.
 17. The device of claim14, wherein the first interaction mode indicates the function to beinvoked in connection with the second input.
 18. The device of claim 17,further comprising a touchscreen, wherein: the second interaction modeis determined to be a touch interaction mode in response to adetermination that a touch input is input to the touchscreen; or thesecond interaction mode is determined to be a gesture interaction modein response to a determination that a gesture is input before a touchinput is input to the touchscreen.
 19. The device of claim 14, whereinthe interface comprises: a widget, the widget including a controlelement, and the control element is configured to respond to a slideevent, and a position of the control element matches a position of ahand of a user.
 20. A computer program product, the computer programproduct being embodied in a non-transitory computer readable storagemedium and comprising computer instructions for: obtaining, by one ormore processors, a first input, the first input being input via a firstinteraction mode; providing, by the one or more processors, aninterface, the interface being displayed based at least in part on thefirst input; obtaining, by the one or more processors, a second input,the second input being input via a second interaction mode, the secondinteraction mode being different from the first interaction mode; andperforming, by the one or more processors, a function corresponding tothe interface, the function being based at least in part on one or moreof the first input, the first interaction mode, the second input, andthe second interaction mode.